An impact review of a Western Australian research translation program

The translation gap between knowledge production and implementation into clinical practice and policy is an ongoing challenge facing researchers, funders, clinicians and policy makers globally. Research generated close to practice and in collaboration with end users is an approach that is recognised as an effective strategy to facilitate an improvement in the relevance and use of health research as well as building research capacity amongst end users. The Research Translation Projects (RTP) program funded by the Western Australian (WA) Department of Health facilitates clinical and academic collaboration through competitive funding of short-term research projects. Its aim is to improve healthcare practice while also finding efficiencies that can be delivered to the WA health system. A mixed methods approach was adopted to evaluate the research impact of the RTP program, at completion of the two-year funding period, across a range of impact domains through the adaptation and application of the Canadian Academy of Health Sciences’ (CAHS) framework for research impact. In addition, further analysis was undertaken to address specific objectives of the RTP program more closely, in particular research capacity building and collaboration and health system Inefficiencies targeted by the program. Social network analysis was applied to assess the extent and growth of collaboration across WA health organisations over time. Results indicated that the ‘bottom up’ approach to research translation has triggered modest, yet positive outcomes across impact domains including advancing knowledge, collaboration and capacity building as well as contributing to changes in policy and practice. Additionally, the projects identified opportunities by which inefficiencies in the health system can be addressed. Further work is required to better understand the pathways by which short-term outcomes can be translated into more long-term impacts and the mechanisms that trigger this process.


Background
The production of quality evidence to inform policy and practice in healthcare requires significant investment. However, implementation and translation of healthcare research into policy While internal evaluation and monitoring of the RTP program has been undertaken, a more formal evaluation has the capacity to better inform ongoing investment. The measurement and assessment of returns on research investment is high on the policy agenda in Australia and across the globe [15]. Additional to accountability upwards to funders and downwards to users, the evaluation of research impact can also contribute to the development and strengthening of research dissemination and translation, inform ongoing funding allocation to bring about desired impact as well as manage and monitor the performance of research programs [16,17]. The aim of this study is to evaluate the research impact of the RTP program across a range of impact domains consistent with the programs focus. Specific objectives of the study are to: (i) assess the research impact of research translation projects at the completion of funding against an adapted version of an existing evaluation framework, (ii) undertake social network analysis of collaboration across the WA health system as a result of the RTP program and (iii) identify sources of inefficiency addressed by projects and the proposed decision-making informed by the project outcomes.

Methods
A mixed methods approach was adopted to evaluate the RTP program based on document analysis. Research impact of the program was assessed along the spectrum of research translation through the adaptation of an existing framework [18]. In addition, further analysis was undertaken to address the objectives of the RTP program more closely, specifically with regard to research capacity building and collaboration as desired outcomes of the RTP program as well as areas of inefficiency targeted by projects funded under the RTP program.

Selection of the evaluation framework
Driven by a policy agenda that demands accountability, reduced waste and a focus on sustainable health systems, research impact has become a topical issue and, as such, there is growing interest in measurement internationally. We drew on the literature to identify a suitable conceptual framework to support the RTP program evaluation. A number of authors have attempted to summarise the use of research evaluation frameworks and tools presented in the literature through comparative studies and systematic reviews, including the "real world" impact of research, rather than just that which is applicable to academia [1,16,19,20]. Cruz Rivera, Kyte [16] reported on twenty-four unique methodological frameworks for assessing health research impact. In recent reviews, both Cruz Rivera, Kyte [16] and Raftery, Hanney [19] have identified the Payback Framework as the most widely applied model for the evaluation of funded health research programs, a finding that is consistent with a review, conducted by Hanney, Buxton [21]. The Payback Framework developed by the Health Economics Research Group at Brunel University was originally designed to assess the return on investment from health service research [22]. The concept of "payback" as the foundation of the work of Buxton and Hanney [22] focuses on the benefit of research across a number of domains, such as knowledge, future research, research use and political and societal benefits. In recent years, the Payback Framework has been applied and adapted in the development of other methodological frameworks [16,19].
The framework chosen here for the assessment of research impact of the RTP program is the Canadian Academy of Health Sciences' (CAHS) framework [18], an adaption of the Payback Framework. The CAHS framework was selected because it provides a menu of defined and published preferred indicators and metrics for each category and subcategory of the framework; a unique advantage not provided by other frameworks. We chose indicators that are generic and could be applied across the four pillars of research outlined in the CAHS, namely biomedical research, applied clinical research, health services and policy research and population and public health research [18]. The adaptability of the CAHS framework across these four pillars of research is important given the heterogeneity of the research projects funded through the RTP program. Additionally, this evaluation of the RTP program is over a relatively short timeframe, with projects being assessed up to the end of a two-year funding period. The CAHS framework considers research impact along a pathway of translation, which allows outcomes that are feasible to assess in the short term to be evaluated. Finally, the CAHS framework was selected based on its previous use within the RTP program, for internal evaluations as well as in program documentation.
The CAHS framework is designed to capture health research impact across five main categories: 1) advancing knowledge, 2) building capacity 3) informing decision-making, 4) health impacts, and 5) broad socio-economic impacts. Given the short timeframe for the evaluation, research projects were only assessed across the first three categories. Research impact in categories 4 and 5 are unlikely to be achieved at the completion of a two-year project. As instructed by the framework developers, the set of indicators and metrics chosen for this evaluation were based on their 'feasibility' and 'attractiveness', primarily on account of data availability, timeliness, and attribution.
A key goal of the RTP program is to facilitate clinical and academic collaboration. Literature suggests that a benefit of clinical and academic collaboration is the development of research capacity amongst the clinical workforce, which has been argued to be fundamental to closing the evidence-practice gap [23]. In addition to using the CAHS framework for measuring research impact, the principles of capacity building proposed by Cooke and colleagues [23][24][25] were applied to assess the contribution of the RTP program to research capacity building across the WA health system. These principles are 1) skill development 2) developing research 'close to practice' 3) building linkages and partnerships 4) appropriate dissemination 5) continuity and sustainability and 6) establishing infrastructure.
In addition to facilitating clinical and academic collaboration the RTP program also seeks to improve efficiencies in the WA health system. Due to the analysis being undertaken in the short term, a framework to capture the types of system inefficiencies at the centre of each project was developed based on the work of Ravaghi, Afshari [26] on hospital inefficiencies and the World Health Organization on technical inefficiencies relating to health system inputs [27]. Four broad types of inefficiencies were identified: 1) inefficient use of procedures, investigation and equipment, 2) irrational use and supply of drugs and blood products, 3) inappropriate or costly workforce mix and 4) inappropriate hospital admissions and length of stay and suboptimal quality of care.

Data sources and extraction
The primary source of data was archival documents of the RTP program including application guidelines, grant applications and completion reports from researchers. Data extraction was undertaken using a systematic examination of completion reports and corresponding grant applications for RTP projects funded between 2011 and 2016. Each grant recipient is required to submit a completion report to provide an account of objectives achieved, whether or not the program/intervention was found to provide efficiencies to the health system and whether the funding had led to other benefits (such as changes in culture, capacity, new collaborations etc.). Completion reports also provide information regarding research outputs such as publications and presentations resulting from the funded research. Given the two-year funding period of the RTP program, the selection of projects funded between 2011 and 2016 provided sufficient time for project completion and submission of final completion reports, with an allowance of time for slippage. Projects were included in the review only where the final reports and corresponding applications were available. For the assessment of collaboration, applications for successfully funded projects were included beyond 2016 up until 2019. Given that the applications and reports are completed by the research teams, the data they contain is selfreported and the data quality is limited to the level to which the applications and reports are completed.

Data analysis
Several approaches were applied to analyse data extracted from the RTP documents, depending on the indicators and metrics used. These included content and thematic analysis and social network analysis. In the first instance, a content analysis against indicators and metrics selected from the CAHS research impact framework was undertaken for descriptive quantitative analysis (i.e. counts). Table 1 identifies the specific indicators and metrics extracted from each document as part of the content analysis.
Guided by Sarre and Cooke's principles of research capacity building [25], further analysis of the collaboration between clinicians and academics was undertaken using social network analysis (SNA) and thematic analysis [25].
The SNA was conducted at the organisation level, identifying the organisations represented within each research team between 2011 and 2019. Organisations represented on each successfully funded project team were coded into categories ( Table 2). Nodes in the network graph are used to represent the organisations while the connections (edges) are weighted (i.e. drawn thicker) based on the number of project collaborations. The data analysis and processing was developed using Python and NetworkX (https://networkx.org/), a module for network analysis, in addition to developing a Web application using a JavaScript library named vis.js (https://visjs.org/) for building interactive network visualisations.
Deductive thematic analysis was also undertaken to illustrate the contribution of the RTP program to capacity building. Direct quotes reflecting the principles of research capacity building developed by Cooke [24] were identified in RTP final reports and coded based on these principles. Finally, sources of inefficiency and proposed solutions were identified from information presented in the RTP final reports and coded based on the adapted framework developed from the work of Ravaghi, Afshari [26] and the World Health Organization [27].

Research impact
Of the 59 projects funded between 2011 and 2016, the required documentation was available for 33 projects for which the research focus spanned across a spectrum of acute hospital-based research to community care, general practice as well as diagnostics and clinical education From a broader point of view the RTPs by nature of the grant guidelines are focused on quality improvement issues. In addition the cost savings perspective means that the research projects are largely quantitative, pre-post or before after studies. Project funding was for a term of two years with a maximum amount of $270 000. Fig 1 provides a summary of the short-term research impacts in terms of the traditional impact measures. The longer-term impacts, beyond 5 years, such as the effects on healthcare and risk factors and contributions to changing health, social and economic wellbeing were beyond the scope of this study. In relation to Advancing knowledge, 60 peer-reviewed publications were reported as having been published as a result of RTP funding, however, upon citation analysis only 16 publications acknowledged this funding. Of these 16, 81% (n = 13) were published in high impact (Q1) journals [28]. Twenty-one projects reported publications in draft at the time of completion. There were 122 conference presentations resulting from the 33 projects, and 10 projects reported media coverage. Other publications noted in the final reports included educational resources and unpublished theses.
Within the category of capacity building, content analysis revealed that across the traditional measure of capacity building, of the 33 projects, six PhD candidates used the projects for research contributing to their doctoral award. Nineteen projects gained additional research funding leveraged off the initial RTP funding, four of which were NHMRC grants.
Finally, in relation to influencing decision making, 14 projects reported a contribution to the implementation of new local practice guidelines or policy within the setting where the project was undertaken. Eight projects reported making contributions either to changes to policy and guidelines beyond the local setting.

Research capacity building and collaboration
Both thematic and social network analysis was undertaken to provide a deeper understanding of the extent of collaboration and research capacity development as a result of the RTP program. Thematic analysis. The review of narratives on research capacity developed as a result of the RTP program reflected capacity building in line with the six principles proposed by Cook (2005) ( Table 3). Additionally, improved collaboration was a noticeable outcome of the RTP program as reported by investigators in terms of both clinical collaboration and research collaboration.
Social network analysis. The extent of the collaboration network generated by the RTP program across the WA health system was depicted using SNA.   (Fig 2). To illustrate the collaboration between academics and service providers, academic organisations are represented as purple nodes and clinical service organisations as orange nodes. For details of organisations included in each group, refer to Table 2. It is important to note that two nodes, identified in green represent consumers or consumer representative organisation. In addition to an overall growth in the collaborative network, Fig 3 distinguishes organisations with a special interest in primary care (red nodes), highlighting their increased involvement (although small) of these organisations in the RTP network over time.

Sources of inefficiency.
An additional feature of the RTP program is its focus on efficiencies and generating cost saving. Report narratives provided by investigators suggested that, as a result of participating in the RTP program, clinicians and researchers developed a sense of resource awareness and an appreciation for undertaking economic evaluation as part of intervention research.

"Being involved in this study has provided internal staff with experience in clinical research and the value of analysis of cost effectiveness and considering new models of care to maximise the health dollar"
"The project has given researchers at the. . . a better understanding of health economics. . ." Of the 33 projects assessed, 67% (n = 22) attempted to address service delivery inefficiencies. The majority of these primarily aimed to reduce inpatient hospital costs related with inappropriate hospital admissions and length of stay and suboptimal quality of care (Table 4). Strategies employed to address these inefficiencies included improved coordination of care, shifting care from inpatient to community or home-based care and implementation of care pathways for inpatient admissions. Six projects (n = 6) attempted to address inefficiencies in relation to tests and procedures, proposing new test or procedure technology that could deliver the same or better outcomes at a lower cost. Three projects focused on inefficiencies relating to the irrational use and supply of drugs or blood products. Examples included the implementation of new guidelines to improve dispensary efficiency and effectiveness, and the trial of an alternative, more cost-effective therapy. Finally, workforce substitution from medical staff to allied health staff was the focus of two projects (n = 2) to address inefficiencies related to an inappropriate or costly workforce mix.

Discussion
This study reports on the assessment of the RTP program against established research impact indicators including advancing knowledge, building capacity and informing decision making in line with the short-term domains of the CAHS framework, leaving the long term domains for future investigation. While common indicators of advancing knowledge and capacity building have been widely used and reported in the literature (e.g. publication count, additional funding etc.), we have applied more novel methods such as SNA to evaluate capacity building and collaboration in greater detail in addition to the traditional approaches.
Based on the findings from this study, the RTP program has made some notable contributions to a range of established research impact categories. Despite over-reporting by investigators, evidenced by the 60 peer review publications reported but only 16 explicitly attributed to the RTP grant funding, the program has demonstrated research output in the form of peer reviewed publications, conference presentations, media appearances and other publications such as educational resources, unpublished thesis and non-academic reports. It is recognised that counting publications is a simple way to measure research output and is a limited indicator of impact without an accepted benchmark for comparison [18]. Given the short time frame over which projects were assessed it was unrealistic to undertake a citation analysis at the completion of funding. In addition to the number of publications, the quality of publication output was of a high standard, as evidenced by the majority of publications appearing in high impact journals [28]. While the RTP program has demonstrated achievement across the 3 short term domains of the CHAS framework the research translation projects were not without issues. Several projects identified problems with recruitment, ability to access data, changes to the practice setting, changes to funding of services and movement of staff away from the study area, all of which impacted on the completion and quality of research.
A key aim of the RTP program is to integrate health professionals more effectively into clinical research activity that addresses clinical needs by providing practical solutions on the frontline of care. There was evidence within the project completion reports that the RTP program led to changes to policy or implementation of new practice guidelines within the local setting of the intervention and in some cases beyond the that setting, which is an illustration of its ability to contribute to evidence-based policy and practice. However, the extent of uptake of such practice guidelines and policy and the sustainability of such changes are not well understood.

Service delivery
Inappropriate ALS � , unnecessary admissions or unnecessary referrals to specialists due to historical and inappropriate practice. Inappropriate service availability. Insufficient guidelines and treatment plans during admission Improving coordination of care and care management processes; Shifting care from hospital inpatient to community/home setting; Implementation of follow up and screening services to reduce hospital admissions and LOS; Implementation of care pathways for inpatient admissions; intervention to improve provider communication and collaboration; patient education; clinician education; implementation of adjunct therapy to reduce LOS n = 22 Inappropriate hospital admissions and length of stay and suboptimal quality of care

Tests and procedures
Misuse or inappropriate use of technology in patient treatment and diagnosis like imaging and lab services due to lack of adopted evidenced-based guidelines and technologies and/or lack of knowledge and skills of health professional. Beyond the traditional measures of research output, the RTP program, with relatively modest funds, appears to have made some contribution to developing research capacity and collaboration across the WA health system. The social network analysis illustrates the breadth of organisations who are collaborating across the WA health system via the RTP program. While there is currently limited involvement of consumer groups and primary care, this appears to be growing over time which is an important finding given the commitment of the WA department of health to collaborate and engage more meaningfully with consumers and primary care through research, evaluation and health service improvement as outlined in the sustainable health review [29]. Increased collaboration across acute and academic organisations with primary care is also beneficial as we attempt to move away from hospital based care to preventative care.
This study has highlighted research capacity building as an added value of undertaking collaborative research. While collaboration is inevitable because of the nature of the projects, research capacity building can be seen as a by-product of the opportunity provided throughout the RTP program to undertake collaborative research. This was evidenced by specific reference to research capacity building principles in the report narratives. In addition, indicators for capacity building from the CAHS framework were also reported on, with the program contributing to a number of higher degrees and supporting the procurement of additional funding. Research capacity building is considered fundamental to closing the evidence-practice gap [23] and advancing both the individual and organisation's ability to conduct, translate and sustain higher quality research and practice [30][31][32]. The development of skills and research knowledge as the first principle of research capacity building was a powerful message that came across throughout the majority of the reports assessed. The importance of programs such as the RTP program is therefore accentuated by the growing concerns for the research capacity of the clinical workforce and the competitiveness of clinician-researchers when it comes to securing funding for quality research [33][34][35].
In addition to increasing research skills and knowledge, collaboration and linkage was a notable benefit reported by investigators as a result of undertaking an RTP. The SNA effectively demonstrated the growth in the RTP collaborative network over time and the balance of collaboration between health service organisations and academic institutions. The notion of building capacity in research and facilitating collaboration is dependent on funding and support opportunities from funding like the RTP program. Given the benefits of clinician engagement in research outlined in this paper, initiatives to develop and facilitate such opportunities should be supported.
The evaluation focus was over a short time period, which meant only the research outputs and impacts that were present at the end of the funding period could be captured and long term impacts which are subject to significant time lags such as health impacts, and broad economic and social impacts where not able to be assessed [16]. While the broad economic impacts were unable to be assessed, we were able to identify the specific sources of inefficiencies that each of the RTPs where addressing to reduce waste in the system. The largest source of inefficiency to have been addressed was in service delivery where inappropriate hospital admissions, length of stay and suboptimal quality of care drive up healthcare costs. Other opportunities that were adopted to promote greater efficiency and a more sustainable heath system were the inefficient use of procedures, investigations and equipment, the irrational use and supply of medicines and blood products as well as inappropriate workforce mix. While the detail of economic analysis presented in the final reports made it difficult to estimate that tangible cost savings had been made, eligibility for RTP funding requires clinicians and researchers to actively seek more efficient ways of delivering healthcare across a range of inefficient practices without compromising health outcomes.
While this study has provided important insights into the contribution of the RTP program to indicators of research impact, capacity building and identifying opportunities for efficiency gains in the WA health system, insight into how research translation projects such as those funded by the RTP program achieve these outcomes has not been explored. Future research on the RTP program is planned, delving into the mechanisms that lead to positive outcomes such as research translation and its longer-term contribution to overall health system sustainability.

Limitations
The completeness of the final reports submitted by investigators varied greatly and, in many cases, completion reports were poorly done. With only 33 out of 59 projects able to be assessed, this in itself reflects waste to the system as there might have been important findings to improve efficiency that were missed. Incomplete reporting also potentially introduces a degree of bias into the study's results. The self-reporting nature of the data collection may also be a limitation with the potential for self-reporting bias and overstating positive outcomes of the research as well as a bias towards completion with only those who felt they were successful bothering to complete the final report The short-term nature of the evaluation is a further limitation to capturing the outputs and impacts that may have arisen over time. It is widely acknowledged that research impacts on health and broader societal and economic outcomes is associated with significant time lags. There is therefore a need for further evaluation beyond the two-year funding period to better understand the extent of theses outcomes and impacts of the RTP program.

Conclusion
The RTP program has contributed to common measures of research impact such as advancing knowledge, building research capacity and collaboration and, to a lesser extent, contributing to few yet worthy changes to policy and practice. These outcomes have been achieved through clinician-initiated projects, in collaboration with academic researchers, and implemented within local health organisations in response to an opportunity to initiate change with a potential to generate efficiencies. This approach to initiating change is consistent with one of the enduring strategies put forward by the recent Western Australia Sustainable Health Review, which reiterates the notion of partnerships with clinicians, consumers and a wide range of partners being needed to develop, test and spread initiatives that deliver better patient care, and are vital to achieving a more sustainable health system [29].
While the research presented here highlights the short term research outputs and contributions to research capacity building and collaboration, it does not explore how these outcomes were achieved or the mechanisms which may contribute to the spread and uptake of research and innovations across the system in the longer term. In addition, a lens that looks beyond the two-year funding period would provide insight into research outcomes that are subject to time lags, such as changes in policy and practice and economic impacts at a system or societal level. Further research is required to better understand the mechanisms triggered by the RTP program to facilitate the spread and uptake of research innovation across the system over time. Understanding the 'how' will enable the recommendation made by the Sustainable Health Review to be implemented more effectively to achieve a more sustainable health system.